Methods and devices for generation and processing of modified audio bitstreams

ABSTRACT

Described herein is a method for generating a modified bitstream on a source device, wherein the method includes the steps of: a) receiving, by a receiver, a bitstream including coded media data; b) generating, by an embedder, payload of additional media data and embedding the payload in the bitstream for obtaining, as an output from the embedder, a modified bitstream including the coded media data and the payload of the additional media data; and c) outputting the modified bitstream to a sink device. Described is further a method for processing said modified bitstream on a sink device. Described are moreover a respective source device and sink device as well as a system of a source device and a sink device and respective computer program products.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to United States Provisional PatentApplication No. 62/887,098, filed Aug. 15, 2019 and European PatentApplication No. 19191920.8, filed Aug. 15, 2019, each of which is herebyincorporated by reference in its entirety.

TECHNOLOGY

The present disclosure relates generally to a method and source devicefor generating a modified bitstream, and more specifically to generatinga modified bitstream by embedding payload of additional media data intoa bitstream including coded media data. The present disclosure alsorelates to a method and sink device for processing said modifiedbitstream.

While some embodiments will be described herein with particularreference to that disclosure, it will be appreciated that the presentdisclosure is not limited to such a field of use and is applicable inbroader contexts.

BACKGROUND

Any discussion of the background art throughout the disclosure should inno way be considered as an admission that such art is widely known orforms part of common general knowledge in the field.

Recent efforts have been made to establish AAC-family based codecs aspart of the Bluetooth (BT) Ecosystem. In this context, the Bluetooth SIGmay define use-cases which require to pass-through bitstreams receivedon, for example, a mobile device (BT source) to connected speakersystems (BT sink). As the data-format used from the service throughoutthe BT endpoint may be coded audio, there is no easy way of conveyingearcons and/or system sounds from the mobile device to the BT speaker.Such earcons may include ringtones, mail notification sounds, key-presssounds and so on.

In currently deployed systems, the audio received from a service isusually decoded in the device into the uncompressed domain (PCM), whereit can be modified and/or mixed. If a pass-through mode of operation isenabled, there may be no easy possibility on the BT-source device inorder to enable and/or mix system sounds into the experience which istransported from BT source to BT sink.

One possibility to overcome this limitation by still enablingpass-through, would be to open an additional link between both devices.However, this setup has the drawback that it needs an additional linkand encoder-decoder pair of a Bluetooth-codec in order to work. Thiscomes with unwanted additional system complexity and burdens both the BTsource and sink device with additional computational complexity, whichmay cost battery as well.

Accordingly, there is an existing need for simultaneous generating andprocessing of earcons and/or system sounds in parallel to the mediacontent via bitstreams, in particular in MPEG-4 audio format.

SUMMARY

In accordance with a first aspect of the present disclosure there isprovided a method for generating a modified bitstream by a sourcedevice. The source device may comprise a receiver, and an embedder. Themethod may include the step of a) receiving, by the receiver, abitstream including coded media data. The method may further include thestep of b) generating, by the embedder, payload of additional media dataand embedding the payload in the bitstream, for obtaining, as an outputfrom the embedder, a modified bitstream including the coded media dataand the payload of the additional media data. And the method may includethe step of c) outputting the modified bitstream to a sink device.

Configured as proposed, the method allows to insert system sounds orEarcons into a bitstream that includes coded media data and that ispassed through by a source device (e.g., a Bluetooth source device, suchas a mobile phone) for a sink device (e.g., a Bluetooth sink device,such as a Bluetooth loudspeaker). This enables to pass systemsounds/Earcons to the sink device without having to establish anadditional connection between the source device and the sink device, andwithout having to decode and re-encode the coded media data. Moreover,the method enables simultaneous processing of the system sounds/Earconsand the coded media data.

In some embodiments, the additional media data may be played back by thesink device.

In some embodiments, the additional media data may be played back by thesource device.

In some embodiments, the coded media data in the received bitstream maybe passed through the source device to the sink device via the modifiedbitstream.

In some embodiments, step b) may further include generating informationon the additional media data and embedding the information in thebitstream.

In some embodiments, the generated information may include one or moreof information on a configuration of the additional media data,information on a presence of the additional media data in the modifiedbitstream, information on a default loudness value, information on avalue for delay alignment, information on delay adaption, information onpanning and information on a type of the additional media data.

In some embodiments, step b) may further include generating informationrelating to a relative gain between the coded media data and theadditional media data and embedding the information relating to therelative gain in the bitstream.

In some embodiments, the modified bitstream may be in MPEG-4 audioformat. Accordingly, system sounds/Earcons can be transmitted to thesink device as part of the coded bitstream when the source device is inpass-through mode for media data that is coded in MPEG-4 audio format,without need to decode and re-encode the media data.

In some embodiments, the generated payload may be embedded in thebitstream by transporting the payload in the (modified) bitstream via anMPEG-4 audio bitstream extension mechanism.

In some embodiments, further the generated information and/or thegenerated information relating to the relative gain may be embedded inthe bitstream by transporting the generated information relating to therelative gain in the (modified) bitstream via the MPEG-4 audio bitstreamextension mechanism.

In some embodiments, the MPEG-4 audio bitstream extension mechanism maybe a data stream element (DSE).

In some embodiments, the data stream element may be located at apredefined position in the modified bitstream and/or may include aspecific instance tag.

In some embodiments, the MPEG-4 audio bitstream extension mechanism maybe a Fill Element (ID_FIL).

In some embodiments, the generated payload may be transported via ageneric extension payload element inside the Fill Element.

In some embodiments, further the generated information and/or thegenerated information relating to the relative gain may be transportedvia the generic extension payload element inside the Fill Element.

In some embodiments, the generated payload may be transported via a newextension payload element inside the Fill Element.

In some embodiments, further the generated information and/or thegenerated information relating to the relative gain may be transportedvia the new extension payload element inside the Fill Element.

In some embodiments, the new extension payload element may include aunique identifier.

In some embodiments, in step c) the modified bitstream may be output tothe sink device via a Bluetooth connection.

In some embodiments, the additional media data may be compressed mediadata or uncompressed media data.

In some embodiments, the uncompressed media data may be PCM datagenerated at the source device.

In some embodiments, the generation of the PCM data may be based on auser input.

In some embodiments, the PCM data may include one or more of earcon-dataand system sounds data.

In some embodiments, the compressed data may be in SBC format or in anaptX-based format.

In accordance with a second aspect of the present disclosure there isprovided a method for processing a modified bitstream by a sink device.The sink device may comprise a receiver, a bitstream parser, a coredecoder and a mixer. The method may include the step of a) receiving, bythe receiver, a modified bitstream including coded media data andpayload of additional media data. The method may further include thestep of b) parsing, by the bitstream parser, the modified bitstream intothe coded media data and the payload of additional media data. Themethod may further include the step of c) core decoding, by the coredecoder, the coded media data to obtain core decoded media data. Themethod may further include the step of d) mixing, by the mixer, the coredecoded media data and the additional media data to obtain an outputsignal. And the method may include the step of e) outputting the outputsignal.

In some embodiments, the modified bitstream may further includeinformation on the additional media data, and the method may furtherinclude, after step a) and before step b), a step of processing themodified bitstream may be based on the information.

In some embodiments, the information may include one or more ofinformation on a configuration of the additional media data, informationon a presence of the additional media data in the modified bitstream,information on a default loudness value, information on a value fordelay alignment, information on delay adaption, information on panningand information on a type of the additional media data.

In some embodiments, the modified bitstream may further includeinformation relating to a relative gain between the coded media data andthe additional media data, and mixing in step d) the core decoded mediadata and the additional media data may be based on the informationrelating to the relative gain.

In some embodiments, the method may further include the step ofprocessing, by a processing unit, the additional media data prior tomixing the core decoded media data and the additional media data.

In some embodiments, processing the additional media data may includeone or more of resampling, delay adaption and loudness processing.

In some embodiments, the additional media data may be compressed mediadata and the method may further include the step of decoding, by adecoder, the compressed media data to obtain decoded additional mediadata, and in step d) the decoded additional media data may be mixed withthe core decoded media data.

In some embodiments, the compressed media data may be in SBC format orin an aptX-based format.

In some embodiments, the additional media data may be uncompressed mediadata, and in step d) the uncompressed additional media data may be mixedwith the core decoded media data.

In some embodiments, the modified bitstream may be in MPEG-4 audioformat.

In some embodiments, the modified bitstream may include an MPEG-4 audiobitstream extension mechanism, and parsing in step b) may further bebased on identifying the MPEG-4 audio bitstream extension mechanism inthe modified bitstream.

In some embodiments, the MPEG-4 audio bitstream extension mechanism maybe a data stream element (DSE) and parsing in step b) may further bebased on identifying a position of the data stream element in themodified bitstream and/or identifying a specific instance tag of thedata stream element.

In some embodiments, the MPEG-4 audio bitstream extension mechanism maybe a Fill Element (ID_FIL) and parsing in step b) may further be basedon identifying an extension payload element inside the Fill Element.

In accordance with a third aspect of the present disclosure there isprovided a source device for generating a modified bitstream. The devicemay include a) a receiver configured to receive a bitstream includingcoded media data. The device may further include b) an embedderconfigured to generate payload of additional media data and to embed thegenerated payload in the bitstream to obtain a modified bitstreamincluding the coded media data and the payload of the additional mediadata. And the device may include c) a control unit configured to outputthe modified bitstream to a sink device.

In some embodiments, the embedder may further be configured to generateinformation on the additional media data and to embed the information inthe bitstream.

In some embodiments, the embedder may further be configured to generateinformation relating to a relative gain between the coded media data andthe additional media data and to embed the information relating to therelative gain in the bitstream.

In some embodiments, the modified bitstream may be in MPEG-4 audioformat.

In some embodiments, the additional media data may be compressed mediadata or uncompressed media data.

In some embodiments, the uncompressed media data may be PCM datagenerated at the source device.

In some embodiments, the generation of the PCM data may be based on auser input.

In some embodiments, the PCM data may include one or more of earcon-dataand system sounds data.

In some embodiments, the compressed media data may be in SBC format orin an aptX-based format.

In accordance with a fourth aspect of the present disclosure there isprovided a sink device for processing a modified bitstream. The devicemay include a) a receiver configured to receive a modified bitstreamincluding coded media data and payload of additional media data. Thedevice may further include b) a bitstream parser configured to parse themodified bitstream into the coded media data and the payload of theadditional media data. The device may further include c) a core decoderconfigured to core decode the coded media data to obtain core decodedmedia data. The device may further include d) a mixer configured to mixthe core decoded media data and the additional media data to obtain anoutput signal. And the device may include e) a control unit configuredto output the output signal.

In some embodiments, the device may further include a processing unitconfigured to process the additional media data prior to mixing the coredecoded media data and the additional media data.

In some embodiments, the additional media data may be uncompressed mediadata, and the mixer may be configured to mix the core decoded media dataand the uncompressed additional media data.

In some embodiments, the additional media data may be compressed mediadata and the device may further include a decoder to decode thecompressed media data to obtain decoded additional media data, and themixer may be configured to mix the core decoded media data and thedecoded additional media data.

In some embodiments, the modified bitstream may further includeinformation on the additional media data, and the sink device may beconfigured to process the modified bitstream based on the information.

In some embodiments, the modified bitstream may further includeinformation relating to a relative gain between the coded media data andthe additional media data, and the mixer may further be configured tomix the core decoded media data and the additional media data based onthe information relating to the relative gain.

In some embodiments, the core decoder may be an MPEG-4 audio decoder.

In accordance with a fifth aspect of the present disclosure there isprovided a system of a source device for generating a modifiedbitstream, wherein the source device includes one or more processorsconfigured to perform a method for generating a modified bitstream onsaid source device and a sink device for processing a modifiedbitstream, wherein the sink device includes one or more processorsconfigured to perform a method for processing a modified bitstream onsaid sink device.

In accordance with a sixth aspect of the present disclosure there isprovided a computer program product comprising a computer-readablestorage medium with instructions adapted to cause the device to carryout a method for generating a modified bitstream on a source device whenexecuted by a device having processing capability.

In accordance with a seventh aspect of the present disclosure there isprovided a computer program product comprising a computer-readablestorage medium with instructions adapted to cause the device to carryout a method for processing a modified bitstream on a sink device whenexecuted by a device having processing capability.

BRIEF DESCRIPTION OF THE DRAWINGS

Example embodiments of the disclosure will now be described, by way ofexample only, with reference to the accompanying drawings in which:

FIG. 1 illustrates a flow diagram of an example of a method forgenerating a modified bitstream on a source device.

FIG. 2 illustrates a flow diagram of a further example of a method forgenerating a modified bitstream on a source device.

FIG. 3 illustrates a flow diagram of an example of a method forprocessing a modified bitstream on a sink device.

FIG. 4 illustrates a flow diagram of a further example of a method forprocessing a modified bitstream on a sink device.

FIG. 5 illustrates a flow diagram of yet a further example of a methodfor processing a modified bitstream on a sink device.

FIG. 6 illustrates a flow diagram of yet a further example of a methodfor processing a modified bitstream on a sink device.

FIG. 7 illustrates an example of a source device for generating amodified bitstream.

FIG. 8 illustrates a further example of a source device for generating amodified bitstream.

FIG. 9 illustrates an example of a sink device for processing a modifiedbitstream.

FIG. 10 illustrates a further example of a sink device for processing amodified bitstream.

FIG. 11 illustrates yet a further example of a sink device forprocessing a modified bitstream.

FIG. 12 illustrates yet a further example of a sink device forprocessing a modified bitstream.

FIG. 13 illustrates an example of a device having one or moreprocessors.

DESCRIPTION OF EXAMPLE EMBODIMENTS Generation and Processing of MPEG-4Audio Modified Bitstreams

In the following, methods and devices will be described for generationand processing of modified MPEG-4 audio bitstreams. Within thisdocument, MPEG-4 audio bitstreams may refer to bitstreams compatiblewith the standard set out in ISO/IEC 14496-3, Coding of audio-visualobjects—Part 3: Audio, and all future editions, revisions and amendmentsthereto (“hereinafter MPEG-4 audio”). By the described methods anddevices, additional media data, being uncompressed or compressed mediadata, may be transported and processed together with coded media datadelivering media content. This may enable to transmit and process earconand/or system sounds data without the additional burden of system- andcomputational complexity in both, source and sink device.

Additionally, it may be possible to control all settings (e.g. level ofsystem sounds, DRC etc.) on both the source device and the sink deviceusing a same mechanism and bitstream variables. This means, a relativegain (or mixing gain) used by an embedder of the source device may beexactly the same as a control parameter used by a mixer of the sinkdevice.

Generating a Modified Bitstream on a Source Device

Referring to the example of FIG. 1, an example of a method forgenerating a modified bitstream on a source device is illustrated. Instep S100 a bitstream including coded media data (to be core decoded bya sink device as detailed below) may be received by a receiver of thesource device. The bitstream including the coded media data may bereceived, for example, from one or more media servers. The coded mediadata may include coded audio data. The coded audio data may be compliantwith the MPEG-4 audio standard. The bitstream may, for example, deliveraudio content to be played back by a sink device connected to the sourcedevice. The source device is not limited. For example, the source devicemay be a mobile device, including one or more of a mobile phone, a smartwatch, a laptop and a tablet pc including convertibles. Referring againto the example of FIG. 1, in step S101, payload of additional media data(i.e. other than and in addition to the coded media data, e.g.additional audio data) may be generated and embedded in the bitstream byan embedder.

The additional media data generated in step S101 is to be mixed with themedia data by (at a side of) the sink device. Advantageously, thisenables mixing of the additional media data with the media data at thesink device side.

The coded media data and the additional media data are different (e.g.type of) media data. For example, as explained below, the coded mediadata and the additional media data may have different format. The codedmedia data and the additional media data include different type of mediacontent. The coded media data may include main audio content, e.g.,music, audio, etc. The additional media data may include secondary audiocontent, e.g., earcons, system sounds, key press sounds, e-mailnotification sounds, etc. The main audio content and the secondary audiocontent may be played back by the sink device.

Alternatively or additionally, the main audio content and the secondaryaudio content may be played back by the source device.

In an embodiment, the coded media data may be played back by the sourcedevice, the sink device or both the source device and the sink device.

In an embodiment, the additional media data may be played back by thesource device, the sink device or both the source device and the sinkdevice.

In an embodiment, the additional media data may be compressed media data(to be decoded by a sink device as detailed below to obtain decodedadditional media data) or uncompressed media data. While theuncompressed media data is not limited, in an embodiment, theuncompressed media data may be PCM data generated at the source device.While the generation of PCM data is also not limited, in an embodiment,the generation of the PCM data may be based on a user input. The userinput may be, for example, a key-press.

Alternatively, or additionally, the generation of the PCM data may befacilitated at the source device independent of a user input, e.g. basedon an email notification. In an embodiment, the PCM data may include oneor more of earcon-data and system sounds data. Also, while the format ofthe compressed media data is not limited, the format may belong to theBluetooth ecosystem. In an embodiment, the compressed media data may bein SBC format or in an aptX-based format.

Referring again to the example of FIG. 1, in step S102, as an outputfrom the embedder, a modified bitstream including the coded media dataand the payload of additional media data may be obtained. In step S103the modified bitstream may then be output to a sink device.

The coded media data in the received bitstream is passed through thesource device to the sink device via the modified bitstream. The codedmedia data as received from the source device is embedded in themodified bitstream and outputted to the sink device.

Advantageously, generating (at the source device) and outputting themodified bitstream to the sink device streamlines mixing of the mediadata with the additional media data at the side of the sink device. Asingle bitstream is used by the source device to embed and transport themedia data and the additional media data to the sink device, thereby asingle channel/connection can be used between the source and the sinkdevice. Furthermore, no additional decoders and/or encoders are presentin the source device to, e.g., decode and re-encode the coded mediadata. No additional mixers are used in the source device to, e.g., mixdecoded coded media data with the additional media data.

In an embodiment, the modified bitstream may be output to the sinkdevice via a Bluetooth connection. In other words, the source device maybe connected to a sink device via a Bluetooth connection. For example,earphones connected to a mobile phone via Bluetooth.

In general, the proposed technique and implementations and examplesthereof described throughout this disclosure involve generating andoutputting a modified bitstream to the sink device facilitatestransmission of the payload of additional media data (e.g., systemsounds/Earcons) to the sink device, e.g. when the source device is in apass-through mode of operation. In other words, the payload ofadditional media data may be generated and embedded into the receivedbitstream including the coded media data to generate the modifiedbitstream without, for example, decoding and re-encoding the coded mediadata. Likewise, the additional media data can be passed to the sinkdevice without need to establish an additional connection between thesource device and the since device. As a further advantage, the proposedtechnique enables simultaneous processing of the additional media dataand the coded media data.

Referring now to the example of FIG. 2, a further example of a methodfor generating a modified bitstream on a source device is illustrated.The example of FIG. 2 differs from the example of FIG. 1 detailed abovein that in step S101 further information on the additional media datamay be generated (e.g. as part of metadata) and embedded in thebitstream (as indicated by the dashed line). In an embodiment, thegenerated information may include one or more of information on aconfiguration of the additional media data, information on a presence ofthe additional media data in the modified bitstream, information on adefault loudness value, information on a value for delay alignment,information on delay adaption, information on panning and information ona type of the additional media data. The generated information may beutilized by the sink device to process the received modified bitstreamas detailed below. The information on the configuration of theadditional media data may, for example, include information on asampling rate which may enable to decide at the side of the sink deviceas to whether the additional media data may need to be resampled priorto mixing. Alternatively, or additionally, the information on theconfiguration of the additional media data may further include, forexample, information on a channel mode which may enable to decide at theside of the sink device if the additional media data may need to bedownmixed/upmixed prior to mixing. The information on a default loudnessvalue may include, for example, a measured loudness as according to oneof the methods specified in ITU-R BS.1770-4. The measured loudness mayenable to decide at the side of the sink device at which level theadditional media data may be mixed with the core decoded media data.Further, the information on the type of the additional media data mayindicate at the side of the sink device as to whether the additionalmedia data may need to be decoded, for example, as to whether theadditional media data may be uncompressed or compressed media data.

Alternatively, or additionally, step S101 may further include generatinginformation relating to a relative gain between the coded media data andthe additional media data (e.g. also as part of metadata) and embeddingthe information relating to the relative gain in the bitstream (asindicated by the dashed line). The relative gain between the coded mediadata and the additional media data may be used by the sink device togenerate an output signal based on the received modified bitstream asdetailed below. In this, source device and sink device may use the samerelative gain value. For example, if the source device is a mobilephone, optimal system sound mixing settings may be automaticallydetermined based on an estimated loudness ratio of the system sounds andcoded media data content. For example, the relative gain value mayinclude a mixing gain value and/or an attenuation gain value which maybe used by the mixer of the sink device. By including this informationinto the bitstream, it can be ensured that the additional media data canbe mixed with the coded media data, after decoding, in an appropriatemanner, and cases in which the additional media data is rendered at aninappropriate sound level compared to the coded media data can beavoided. In general, by including the information on the additionalmedia data into the bitstream, the source device is given control overthe processing (e.g., including rendering) of the additional media dataat the sink device. This ensures appropriate rendering of the additionalmedia data at the sink device.

While the format of the modified bitstream is not limited, in anembodiment, the modified bitstream may be in MPEG-4 audio format (i.e.MPEG-4 audio standard compliant bitstream). Referring to the example ofFIG. 1, in an embodiment, the generated payload may be transported inthe modified bitstream via an MPEG-4 audio bitstream extensionmechanism. Referring to the example of FIG. 2, additionally, in anembodiment, further the generated information and/or the generatedinformation relating to the relative gain may be transported in themodified bitstream via the MPEG-4 audio bitstream extension mechanism.

As an example, the following syntax may be used in order to embed thegenerated payload of the additional media data, the generatedinformation on the additional media data and/or the generatedinformation relating to the relative gain into the modified bitstream ofMPEG-4 audio format.

Syntax of system_sound_info( )

Top level syntactic element to convey the generated payload of theadditional media data, the generated information (information on theadditional media data) and/or the generated information relating to therelative gain.

system_sound_info( ) {  // Configuration  system_sound_config_present;1 bslbf  if(system_sound_config_present) {   static config =system_sound_config( )  }  // Payload  system_sound_payload_present;1 bslbf  if(system_sound_payload_present) {  system_sound_payload(config)  } }Semantics of system_sound_info( )

Element Description system_sound_config_present If set,system_sound_config is present in the streamsystem_sound_payload_present If set, system_sound_payload is present inthe streamSyntax of system_sound_config( )

system_sound_config( ) {  num_sys_sounds; 8 uimsbf  for(s = 0; s <=num_sys_sounds; ++s) {   sys_sound_id[s]; 8 bslbf   sys_sound_type[s];4 bslbf   reserved; 1 bslbf   sys_sound_panning[s]; 8 uimsbf  sys_sound_gain_present; 1 bslbf   if(sys_sound_gain_present) {   sys_sound_gain[s] 8 uimsbf   }   sys_sound_loudness_present; 1 bslbf  if(sys_sound_loudness_present) {    sys_sound_loudness[s]; 8 uimsbf  }   sys_sound_attenuation_gain_present; 1 bslbf  if(sys_sound_attenuation_gain_present) {   sys_sound_attenuation_gain[s]; 8 uimsbf   }  switch(sys_sound_type[s]) {   case SYS_PCM:    pcm_data_config( );   break;   case SYS_SBC:    sbc_data_config( ); // Placeholder: Notdefined here    break;   case SYS_APTX:    aptx_data_config( ); //Placeholder: Not defined here    break;   default:    // Not supported   break;   }  } }Semantics of system_sound_config( )Syntax of system_sound_payload( )

system_sound_payload(config) {  // config of type system_sound_config for(s = 0; s <= config.num_sys_sounds; ++s) {   if(!sys_sound_active) {1 bslbf    continue;   }   switch(config.sys_sound_type[s]) {   caseSYS_PCM:    pcm_data_payload( );    break;   case SYS_SBC:   sbc_data_payload( ); // Placeholder: Not defined here    break;  case SYS_APTX:    aptx_data_payload( ); // Placeholder: Not definedhere    break;   default:    break;   }  } }Semantics of system_sound_payload( )

Element Description sys_sound_active System sound is active and shall bedecodedSyntax of pcm_data_config( )

pcm_data_config( ) {  pcm_num_channels; 8 uimsbf  pcm_endianness; 1bslbf  pcm_align_audio; 1 bslbf  reserved; 1 bslbf pcm_sample_rate_index; 5 bslbf  if(0x1f == pcm_sample_rate_index) {  pcm_sample_rate; 24 uimsbf  }  pcm_frame_size_index; 3 bslbf reserved; 1 bslbf  if(0x06 == pcm_frame_size_index) {   pcm_frame_size;16 uimsbf  }  pcm_data_format_index; 4 bslbf }Semantics of pcm_data_config( )Syntax of pcm_data_payload( )

pcm_data_payload( ) {  for(i = 0; i < pcm_frame_size; ++i) { (NOTE 1)  for(ch = 0; ch < pcm_num_channels) {    pcm_sample; bps bslbf (NOTE 2)  }  } } NOTE 1: {grave over ( )}pcm_frame_size{grave over ( )} iseither derived from {grave over ( )}pcm_frame_size_index{grave over ( )}or set directly in {grave over ( )}pcm_data_config{grave over ( )} NOTE2: Bits per sample (bps) can be derived from {grave over( )}pcm_data_format_index{grave over ( )}

Semantics of pcm_data_payload( )

TABLE 3 Semantics of pcm_data_payload( ) Element Description pcm_samplePCM sample, either IEEE 754 (floating point) or two's complement

There are at least two independent extension mechanisms provided by theMPEG-4 audio standard. Either a data stream element (ID_DSE) orextension_payload(int) may be used to transport the generated payload ofadditional media data, the generated information and/or the generatedinformation relating to the relative gain. In an embodiment, the MPEG-4audio bitstream extension mechanism may be a data stream element (DSE).Data stream elements in MPEG-4 audio standard are generally used totransport metadata and not payload of media data (audio content). In anembodiment, the data stream element (DSE) in the MPEG-4 audio bitstreamis used to transport the generated payload of additional media data.Therefore an existing Data stream element in MPEG-4 bitstream is used totransport payload of media data (e.g. earcons) instead and/or inaddition to metadata. An example syntax is given below.

data_stream_element( ) {  element_instance_tag; 4 uimsbf data_byte_align_flag; 1 uimsbf  reserved; 3 uimsbf  cnt = count;8 uimsbf  if (cnt == 255)   cnt += esc_count; 8 uimsbf  if(data_byte_align_flag)   byte_alignment( );  for (i = 0; i < cnt; i++)  data_stream_byte[element_instance_tag][i]; 8 uimsbf }

Element Description element_instance_tag A number to identify the datastream element. Must be handled with care if multiple DSEs are presentcnt Size of the payload in bytes data_stream_byte Containssystem_sound_info

In an embodiment, the data stream element may be located at a predefinedposition in the modified bitstream and/or may include a specificinstance tag.

Alternatively, in an embodiment, the MPEG-4 audio bitstream extensionmechanism may be a Fill Element (ID_FIL). MPEG-4 audio defines anotherextension mechanism with the top level syntactic elementextension_payload(int). For non-resilience payloads,extension_payload(int) may be contained inside a Fill Element (ID_FIL).

fill_element( ) {  cnt = count; 4 uimsbf  if (cnt == 15)   cnt +=esc_count − 1; 8 uimsbf  while (cnt > 0) {   cnt −=extension_payload(cnt);  } }

In an embodiment, the generated payload may be transported via a genericextension payload element inside the Fill Element. Additionally, in anembodiment, further the generated information (information on theadditional media data) and/or the generated information relating to therelative gain may be transported via the generic extension payloadelement inside the Fill Element. For example, for generic data, theextension_type EXT_DATA_ELEMENT may be used.

Alternatively, in an embodiment, the generated payload may betransported via a new extension payload element inside the Fill Element.Additionally, in an embodiment, further the generated information(information on the additional media data) and/or the generatedinformation relating to the relative gain may be transported via the newextension payload element inside the Fill Element.

Fill elements in MPEG-4 audio standard are generally used to transportmetadata and not payload of media data (audio content). In anembodiment, the data stream element (DSE) in the MPEG-4 audio bitstreamis used to transport the generated payload of additional media data(e.g. as indicated above in a generic extension payload element or in anew extension payload element inside the Fill Element). Therefore anexisting Fill Element in MPEG-4 bitstream is used to transport payloadof media data (e.g. earcons) alternatively and/or additionally tometadata.

New extension payload types (extension_type) for the syntactic elementsystem_sound_info( ) may be defined. The syntax of 14496-3, Table 4.59:“Syntax of extension_payload” may be extended as follows:

No. of Syntax bits Mnemonic extensionpayload(cnt) {  extensiontype; 4uimsbf  align = 4;  switch( extension_type ) {   [...]   caseEXT_SYS_SOUND_INFO:    return system_sound_info( );   [...]  } } Note 1:id_aac is the id_syn_ele of the corresponding AAC element (ID_SCE orID_CPE) or ID_SCE in case of CCE. Note 2: The extension_payload( )included here shall not have extension_type == EXT_DATA_LENGTH.

In an embodiment, the new extension payload element may include a uniqueidentifier. The unique identifier may be, for example,EXT_SYS_SOUND_INFO.

Using the above techniques, the additional media data and theinformation on the additional media data can be readily embedded into anexisting MPEG-4 compliant bitstream that is passed-through by the sourcedevice.

Processing a Modified Bitstream on a Sink Device

Referring now the example of FIG. 3, an example of a method forprocessing a modified bitstream on a sink device is illustrated. In stepS200, a modified bitstream including coded media data (e.g. as receivedby the source device) and payload of additional media data may bereceived by a receiver of a sink device.

In step S201, the received modified bitstream may be parsed, by abitstream parser, into the coded media data and the payload ofadditional media data. While the format of the modified bitstream is notlimited, the modified bitstream may be in MPEG-4 audio format (i.e.MPEG-4 audio standard compliant bitstream). Referring to the example ofFIG. 4, in an embodiment, the modified bitstream may include an MPEG-4audio bitstream extension mechanism, and parsing in step S201 mayfurther be based on identifying the MPEG-4 audio bitstream extensionmechanism in the modified bitstream. In an embodiment, the MPEG-4 audiobitstream extension mechanism may be a data stream element (DSE) andparsing in step S201 may further be based on identifying a position ofthe data stream element in the modified bitstream and/or identifying aspecific instance tag of the data stream element.

Alternatively, in an embodiment, the MPEG-4 audio bitstream extensionmechanism may be a Fill Element (ID_FIL) and parsing in step S201 mayfurther be based on identifying an extension payload element inside theFill Element.

Referring to the example of FIG. 3 or 4, in step S202, the coded mediadata may be core decoded by a core decoder.

In step S203, the core decoded media data may be mixed with theadditional media data, by a mixer, to obtain an output signal.

In step 204, the obtained output signal may then be output by a controlunit.

Referring now to the example of FIG. 5, a further example of a methodfor processing a modified bitstream on a sink device is illustrated. Inthe example of FIG. 5, in an embodiment, the modified bitstream mayfurther include information on the additional media data, and processingthe modified bitstream by the sink device may be based on theinformation. In an embodiment, the information may include one or moreof information on a configuration of the additional media data,information on a presence of the additional media data in the modifiedbitstream, information on a default loudness value, information on avalue for delay alignment, information on delay adaption, information onpanning and information on a type of the additional media data.

In the example of FIG. 5, the method may further include the step S202 aof processing, by a processing unit, the additional media data prior tomixing the core decoded media data and the additional media data. Instep S203, the core decoded media data are then mixed with the processedadditional media data. In an embodiment, processing the additional mediadata may include one or more of resampling, delay adaption and loudnessprocessing. The processing of the additional media data may be guided by(based on) the generated information included in the modified bitstreamand provided in step S201 by the bitstream parser (as indicated by thedashed line). For example, resampling may be guided by information on asampling rate of the additional media data which may be included in theinformation on the configuration of the additional media data.Alternatively, or additionally, processing the additional media data mayinclude downmixing/upmixing the additional media data which may beguided by information on a channel mode which may be included in theinformation on the configuration of the additional media data. Loudnessprocessing may be guided, for example, by information on a defaultloudness value which may include a measured loudness as according to oneof the methods specified in ITU-R BS.1770-4.

Alternatively, or additionally, in an embodiment, the modified bitstreammay further include information relating to a relative gain between thecoded media data and the additional media data, and mixing in step S203the core decoded media data and the additional media data, which mayhave optionally been processed prior to mixing to obtain processedadditional media data (as indicated by the dotted lines), may be basedon the information relating to the relative gain. In this, the sinkdevice may use the relative gain value generated at the source device.In other words, both, source device and sink device may use the samerelative gain value. The relative gain value may also further bemodified via a user input in the mixing step S203. The relative gainvalue may include a mixing gain and/or an attenuation gain value.

The information relating to the relative gain may be provided in stepS201 by the bitstream parser as indicated by the dashed line. Theinformation on the additional media data and alternatively, oradditionally, the information relating to the relative gain value may beembedded in the modified bitstream in the system_sound_config top levelsyntax element as described above.

In an embodiment, the additional media data may be compressed media dataor uncompressed media data.

Referring now to the example of FIG. 6, in an embodiment, the additionalmedia data may be compressed media data and the method may furtherinclude the step S205 of decoding, by a decoder, the compressed mediadata to obtain decoded additional media data, and in step S203 thedecoded additional media data may be mixed with the core decoded mediadata. If the method includes further a step of processing the additionalmedia data, S202 a, in case of compressed media data, the decodedadditional media data may optionally be subjected to processing in stepS202 a, as detailed above and indicated by the dotted lines, prior tomixing with the core decoded media data. While the format of thecompressed media data is not limited, the format may belong to theBluetooth ecosystem. In an embodiment, the compressed media data may bein SBC format or in an aptX-based format. Also in case of compressedmedia data, processing in step S202 a may be guided by (based on)generated information on the additional media data included in themodified bitstream and provided in step S201 as indicated by the dashedline.

Alternatively, or additionally, mixing in step S203 the core decodedmedia data and the decoded additional media data, which may haveoptionally been processed prior to mixing, may also be based oninformation relating to a relative gain as detailed above. Theinformation relating to the relative gain may also be provided in stepS201 as indicated by the dashed line.

Alternatively, in an embodiment, the additional media data may beuncompressed media data, and in step S203 the uncompressed additionalmedia data may be mixed with the core decoded media data in accordancewith the example illustrated in FIG. 3 or FIG. 4. In accordance with theexample of FIG. 5, the uncompressed media data may also optionally besubjected to processing in step S202 a, as detailed above, prior tomixing with the core decoded media data. In an embodiment, theuncompressed media data may be PCM data generated at the source device.In an embodiment, the PCM data may include one or more of earcon-dataand system sounds data.

Source Device

Referring now to the example of FIG. 7, an example of a source devicefor generating a modified bitstream is illustrated. The source device100 may include a receiver 102 configured to receive a bitstreamincluding coded media data 101. The bitstream may be received, forexample, from one or more media servers. The bitstream may, for example,deliver audio content to be played back by a sink device connected tothe source device 100. The source device 100 is not limited. Forexample, the source device 100 may be a mobile device including one ormore of a mobile phone, a smart watch, a laptop and a tablet pcincluding convertibles.

In an embodiment, the coded media data may be played back by the sourcedevice, the sink device or both the source device and the sink device.In an embodiment, the additional media data may be played back by thesource device, the sink device or both the source device and the sinkdevice.

The source device 100 may further include an embedder 104 configured togenerate payload of additional media data 103 and to embed the generatedpayload in the bitstream to obtain a modified bitstream including thecoded media data 101 and the payload of additional media data 103. Inthis, a modified bitstream may also be generated even if the sourcedevice is operated in pass-through mode. The source device 100 may beconfigured to pass through the coded media data to the sink device viathe modified bitstream. The coded media data as received from the sourcedevice is embedded in the modified bitstream and outputted to the sinkdevice.

Advantageously, generating (at the source device) and outputting themodified bitstream to the sink device streamlines mixing of the mediadata with the additional media data at the side of the sink device. Asingle bitstream is used by the source device to embed and transport themedia data and the additional media data to the sink device, thereby asingle channel/connection can be used between the source and the sinkdevice. Furthermore, no additional decoders and/or encoders are presentin the source device to, e.g., decode and re-encode the coded mediadata. No additional mixers are used in the source device to, e.g., mixdecoded coded media data with the additional media data.

While the format of the modified bitstream is not limited, in anembodiment, the modified bitstream may be in MPEG-4 audio format (i.e.MPEG-4 audio standard compliant bitstream). In an embodiment, theembedder may be configured to transport the generated payload via anMPEG-4 audio bitstream extension mechanism in the modified bitstream asdetailed above.

In an embodiment, the additional media data 103 may be compressed mediadata or uncompressed media data. While the uncompressed media data isnot limited, in an embodiment, the uncompressed media data may be PCMdata generated at the source device 100. While the generation of PCMdata is not limited, in an embodiment, the generation of the PCM datamay be based on a user input. The user input may be, for example, akey-press. The source device 100 may then include a user interface whichmay be, for example, a touch display. Alternatively, or additionally thesource device 100 may further include at least a keyboard. However, thegeneration of the PCM data may also be facilitated at the source device100 independent of a user input, e.g. based on an email notification. Inan embodiment, the PCM data may include one or more of earcon-data andsystem sounds data. The PCM data may be stored in the source device 100.

While the compressed media data is not limited, the format of thecompressed media data may belong to the Bluetooth ecosystem. In anembodiment, the compressed media data may be in SBC format or in anaptX-based format. The compressed media data may be generated at thesource device 100. The compressed media data may be stored in the sourcedevice 100. Alternatively, or additionally, the compressed media datamay be received via the receiver 102.

Referring now to the example of FIG. 8, in an embodiment, the embedder104 may further be configured to generate information 106 on theadditional media data 103 and to embed the generated information 106 inthe bitstream (as indicated by the dashed line). Alternatively, oradditionally, in an embodiment, the embedder 104 may further beconfigured to generate information 106 relating to a relative gainbetween the coded media data 101 and the additional media data 103 andto embed the information 106 relating to the relative gain in thebitstream (as also indicated by the dashed line). In an embodiment, theembedder may further be configured to transport the generatedinformation and/or the generated information relating to the relativegain via the MPEG-4 audio bitstream extension mechanism in the modifiedbitstream as detailed above.

The source device 100 may further include a control unit 105 configuredto output the modified bitstream to a sink device.

Sink Device

Referring now to the example of FIG. 9, an example of a sink device forprocessing a modified bitstream is illustrated. The sink device 200 mayinclude a receiver 201 configured to receive a modified bitstreamincluding coded media data and payload of additional media data. Thesink device 200 is not limited. The sink device 200 may be a peripheraldevice to the source device, for example, earphones. The sink device 200and the source device may be connected via Bluetooth. The sink device200 may be configured to play back audio content included in themodified bitstream. For example, the sink device 200 may be configuredto play back the coded media data or the additional media data or boththe coded media data and the additional media data.

The sink device 200 may further include a bitstream parser 202configured to parse the modified bitstream into the coded media data andthe payload of the additional media data. While the format of themodified bitstream is not limited, the modified bitstream may be inMPEG-4 audio format (i.e. MPEG-4 audio standard compliant bitstream). Asalready detailed above, in an embodiment, the bitstream parser 202 maybe configured to parse the bitstream based on identifying an MPEG-4audio bitstream extension mechanism in the modified bitstream.

The sink device 200 may further include a core decoder 203 configured tocore decode the coded media data to obtain core decoded media data. Inan embodiment, the core decoder 203 may be an MPEG-4 audio decoder(compliant with the MPEG-4 audio standard as detailed above).

The sink device 200 may further include a mixer 204 configured to mixthe core decoded media data and the additional media data to obtain anoutput signal. In an embodiment, the additional media data may becompressed media data or uncompressed media data. In an embodiment, theadditional media data may be uncompressed media data, and the mixer 204may be configured to mix the core decoded media data and theuncompressed additional media data.

Referring now to the example of FIG. 10, the modified bitstream mayfurther include information on the additional media data, and the sinkdevice 200 may be configured to process the modified bitstream based onthe information. In an embodiment, the sink device 200 may furtherinclude a processing unit 206 configured to process the additional mediadata prior to mixing the core decoded media data and the additionalmedia data in the mixer 204. In an embodiment, processing the additionalmedia data by the processing unit 206 may include one or more ofresampling, delay adaption and loudness processing. Processing theadditional media data by the processing unit 206 may be guided by (basedon) the information on the additional media data as detailed above. Theinformation on the additional media data may be provided by thebitstream parser 202 to the processing unit 206 as indicated by thedashed line. In case of the additional media data being uncompressedmedia data, the uncompressed media data may be directly processed afterthe modified bitstream has been parsed in the bitstream parser 202 andprior to entering the mixer 204.

Alternatively, or additionally, in an embodiment, the modified bitstreammay further include information relating to a relative gain between thecoded media data and the additional media data, and the mixer 204 mayfurther be configured to mix the core decoded media data and theadditional media data, which may optionally have been processed prior tomixing (as indicated by the dotted lines), based on the informationrelating to the relative gain as described above. The informationrelating to the relative gain may be provided to the mixer 204 by thebitstream parser 202 as indicated by the dashed line.

In an embodiment, the additional media data may be compressed mediadata. Referring now to the example of FIG. 11, the sink device 200 mayfurther include a decoder 207 configured to decode the compressed mediadata to obtain decoded additional media data, and the mixer 204 may beconfigured to mix the core decoded media data and the decoded additionalmedia data. Alternatively or additionally, the modified bitstream mayfurther include information relating to a relative gain between thecoded media data and the additional media data, and the mixer 204 mayfurther be configured to mix the core decoded media data and the decodedadditional media data based on the information relating to the relativegain as described above. The information relating to the relative gainmay be provided to the mixer 204 by the bitstream parser 202 asindicated by the dashed line.

Referring further to the example of FIG. 12, the decoded additionalmedia data may further optionally be processed by the processing unit206 prior to mixing with the core decoded media data in the mixer 204(as indicated by the dotted lines). Processing the decoded additionalmedia data by the processing unit 206 may be guided by (based on)information on the additional media data which may be included in themodified bitstream as detailed above. The information on the additionalmedia data may be provided by the bitstream parser 202 to the processingunit 206 as indicated by the dashed line.

Referring to the examples of FIGS. 9 to 12, the sink device 200 mayfurther include a control unit 205 configured to output the outputsignal. For example, the output signal may be output via earphones to auser of the sink device 200.

The above described methods may be implemented individually on the abovedescribed devices. The above described devices capable of performing theabove described methods may also form a respective system. Devices 300,as referred to herein, may include one or more processors 301, 302 asillustrated by way of example in FIG. 13. Alternatively, oradditionally, the above described methods may also be implemented as acomputer program product comprising a computer-readable storage mediumwith instructions adapted to cause the device 300 to carry out saidmethods when executed on a device 300 having processing capability 301,302.

Interpretation

Unless specifically stated otherwise, as apparent from the followingdiscussions, it is appreciated that throughout the disclosurediscussions utilizing terms such as “processing,” “computing,”“calculating,” “determining”, analyzing” or the like, refer to theaction and/or processes of a computer or computing system, or similarelectronic computing devices, that manipulate and/or transform datarepresented as physical, such as electronic, quantities into other datasimilarly represented as physical quantities.

In a similar manner, the term “processor” may refer to any device orportion of a device that processes electronic data, e.g., from registersand/or memory to transform that electronic data into other electronicdata that, e.g., may be stored in registers and/or memory. A “computer”or a “computing machine” or a “computing platform” may include one ormore processors.

The methodologies described herein are, in one example embodiment,performable by one or more processors that accept computer-readable(also called machine-readable) code containing a set of instructionsthat when executed by one or more of the processors carry out at leastone of the methods described herein. Any processor capable of executinga set of instructions (sequential or otherwise) that specify actions tobe taken are included. Thus, one example is a typical processing systemthat includes one or more processors. Each processor may include one ormore of a CPU, a graphics processing unit, and a programmable DSP unit.The processing system further may include a memory subsystem includingmain RAM and/or a static RAM, and/or ROM. A bus subsystem may beincluded for communicating between the components. The processing systemfurther may be a distributed processing system with processors coupledby a network. If the processing system requires a display, such adisplay may be included, e.g., a liquid crystal display (LCD) or acathode ray tube (CRT) display. If manual data entry is required, theprocessing system also includes an input device such as one or more ofan alphanumeric input unit such as a keyboard, a pointing control devicesuch as a mouse, and so forth. The processing system may also encompassa storage system such as a disk drive unit. The processing system insome configurations may include a sound output device, and a networkinterface device. The memory subsystem thus includes a computer-readablecarrier medium that carries computer-readable code (e.g., software)including a set of instructions to cause performing, when executed byone or more processors, one or more of the methods described herein.Note that when the method includes several elements, e.g., severalsteps, no ordering of such elements is implied, unless specificallystated. The software may reside in the hard disk, or may also reside,completely or at least partially, within the RAM and/or within theprocessor during execution thereof by the computer system. Thus, thememory and the processor also constitute computer-readable carriermedium carrying computer-readable code. Furthermore, a computer-readablecarrier medium may form, or be included in a computer program product.

In alternative example embodiments, the one or more processors operateas a standalone device or may be connected, e.g., networked to otherprocessor(s), in a networked deployment, the one or more processors mayoperate in the capacity of a server or a user machine in server-usernetwork environment, or as a peer machine in a peer-to-peer ordistributed network environment. The one or more processors may form apersonal computer (PC), a tablet PC, a Personal Digital Assistant (PDA),a cellular telephone, a web appliance, a network router, switch orbridge, or any machine capable of executing a set of instructions(sequential or otherwise) that specify actions to be taken by thatmachine.

Note that the term “machine” shall also be taken to include anycollection of machines that individually or jointly execute a set (ormultiple sets) of instructions to perform any one or more of themethodologies discussed herein.

Thus, one example embodiment of each of the methods described herein isin the form of a computer-readable carrier medium carrying a set ofinstructions, e.g., a computer program that is for execution on one ormore processors, e.g., one or more processors that are part of webserver arrangement. Thus, as will be appreciated by those skilled in theart, example embodiments of the present disclosure may be embodied as amethod, an apparatus such as a special purpose apparatus, an apparatussuch as a data processing system, or a computer-readable carrier medium,e.g., a computer program product. The computer-readable carrier mediumcarries computer readable code including a set of instructions that whenexecuted on one or more processors cause the processor or processors toimplement a method.

Accordingly, aspects of the present disclosure may take the form of amethod, an entirely hardware example embodiment, an entirely softwareexample embodiment or an example embodiment combining software andhardware aspects. Furthermore, the present disclosure may take the formof carrier medium (e.g., a computer program product on acomputer-readable storage medium) carrying computer-readable programcode embodied in the medium.

The software may further be transmitted or received over a network via anetwork interface device. While the carrier medium is in an exampleembodiment a single medium, the term “carrier medium” should be taken toinclude a single medium or multiple media (e.g., a centralized ordistributed database, and/or associated caches and servers) that storethe one or more sets of instructions. The term “carrier medium” shallalso be taken to include any medium that is capable of storing, encodingor carrying a set of instructions for execution by one or more of theprocessors and that cause the one or more processors to perform any oneor more of the methodologies of the present disclosure. A carrier mediummay take many forms, including but not limited to, non-volatile media,volatile media, and transmission media. Non-volatile media includes, forexample, optical, magnetic disks, and magneto-optical disks. Volatilemedia includes dynamic memory, such as main memory. Transmission mediaincludes coaxial cables, copper wire and fiber optics, including thewires that comprise a bus subsystem. Transmission media may also takethe form of acoustic or light waves, such as those generated duringradio wave and infrared data communications. For example, the term“carrier medium” shall accordingly be taken to include, but not belimited to, solid-state memories, a computer product embodied in opticaland magnetic media; a medium bearing a propagated signal detectable byat least one processor or one or more processors and representing a setof instructions that, when executed, implement a method; and atransmission medium in a network bearing a propagated signal detectableby at least one processor of the one or more processors and representingthe set of instructions.

It will be understood that the steps of methods discussed are performedin one example embodiment by an appropriate processor (or processors) ofa processing (e.g., computer) system executing instructions(computer-readable code) stored in storage. It will also be understoodthat the disclosure is not limited to any particular implementation orprogramming technique and that the disclosure may be implemented usingany appropriate techniques for implementing the functionality describedherein. The disclosure is not limited to any particular programminglanguage or operating system.

Reference throughout this disclosure to “one example embodiment”, “someexample embodiments” or “an example embodiment” means that a particularfeature, structure or characteristic described in connection with theexample embodiment is included in at least one example embodiment of thepresent disclosure. Thus, appearances of the phrases “in one exampleembodiment”, “in some example embodiments” or “in an example embodiment”in various places throughout this disclosure are not necessarily allreferring to the same example embodiment. Furthermore, the particularfeatures, structures or characteristics may be combined in any suitablemanner, as would be apparent to one of ordinary skill in the art fromthis disclosure, in one or more example embodiments.

As used herein, unless otherwise specified the use of the ordinaladjectives “first”, “second”, “third”, etc., to describe a commonobject, merely indicate that different instances of like objects arebeing referred to and are not intended to imply that the objects sodescribed must be in a given sequence, either temporally, spatially, inranking, or in any other manner.

In the claims below and the description herein, any one of the termscomprising, comprised of or which comprises is an open term that meansincluding at least the elements/features that follow, but not excludingothers. Thus, the term comprising, when used in the claims, should notbe interpreted as being limitative to the means or elements or stepslisted thereafter. For example, the scope of the expression a devicecomprising A and B should not be limited to devices consisting only ofelements A and B. Any one of the terms including or which includes orthat includes as used herein is also an open term that also meansincluding at least the elements/features that follow the term, but notexcluding others. Thus, including is synonymous with and meanscomprising.

It should be appreciated that in the above description of exampleembodiments of the disclosure, various features of the disclosure aresometimes grouped together in a single example embodiment, Fig., ordescription thereof for the purpose of streamlining the disclosure andaiding in the understanding of one or more of the various inventiveaspects. This method of disclosure, however, is not to be interpreted asreflecting an intention that the claims require more features than areexpressly recited in each claim. Rather, as the following claimsreflect, inventive aspects lie in less than all features of a singleforegoing disclosed example embodiment. Thus, the claims following theDescription are hereby expressly incorporated into this Description,with each claim standing on its own as a separate example embodiment ofthis disclosure.

Furthermore, while some example embodiments described herein includesome but not other features included in other example embodiments,combinations of features of different example embodiments are meant tobe within the scope of the disclosure, and form different exampleembodiments, as would be understood by those skilled in the art. Forexample, in the following claims, any of the claimed example embodimentscan be used in any combination.

In the description provided herein, numerous specific details are setforth. However, it is understood that example embodiments of thedisclosure may be practiced without these specific details. In otherinstances, well-known methods, structures and techniques have not beenshown in detail in order not to obscure an understanding of thisdescription.

Thus, while there has been described what are believed to be the bestmodes of the disclosure, those skilled in the art will recognize thatother and further modifications may be made thereto without departingfrom the spirit of the disclosure, and it is intended to claim all suchchanges and modifications as fall within the scope of the disclosure.For example, any formulas given above are merely representative ofprocedures that may be used. Functionality may be added or deleted fromthe block diagrams and operations may be interchanged among functionalblocks. Steps may be added or deleted to methods described within thescope of the present disclosure.

Enumerated example embodiments (EEEs) of the present disclosure arelisted below.

A-EEE 1. A method for generating a modified bitstream on a sourcedevice, wherein the method includes the steps of:

-   -   a) receiving, by a receiver, a bitstream including coded media        data;    -   b) generating, by an embedder, payload of additional media data        and embedding the payload in the bitstream for obtaining, as an        output from the embedder, a modified bitstream including the        coded media data and the payload of the additional media data;        and    -   c) outputting the modified bitstream to a sink device.

A-EEE 2. Method according to A-EEE 1, wherein step b) further includesgenerating information on the additional media data and embedding theinformation in the bitstream.

A-EEE 3. Method according to A-EEE 2, wherein the generated informationincludes one or more of information on a configuration of the additionalmedia data, information on a presence of the additional media data inthe modified bitstream, information on a default loudness value,information on a value for delay alignment, information on delayadaption, information on panning and information on a type of theadditional media data.

A-EEE 4. Method according to any of A-EEEs 1 to 3, wherein step b)further includes generating information relating to a relative gainbetween the coded media data and the additional media data and embeddingthe information relating to the relative gain in the bitstream.

A-EEE 5. Method according to any of A-EEEs 1 to 4, wherein the modifiedbitstream is in MPEG-4 audio format.

A-EEE 6. Method according to A-EEE 5, wherein the generated payload isembedded in the bitstream by transporting the payload in the modifiedbitstream via an MPEG-4 audio bitstream extension mechanism.

A-EEE 7. Method according to A-EEE 6, wherein further the generatedinformation and/or the generated information relating to the relativegain are embedded in the bitstream by transporting the generatedinformation and/or the generated information relating to the relativegain in the modified bitstream via the MPEG-4 audio bitstream extensionmechanism.

A-EEE 8. Method according to A-EEE 6 or 7, wherein the MPEG-4 audiobitstream extension mechanism is a data stream element (DSE).

A-EEE 9. Method according to A-EEE 8, wherein the data stream element islocated at a predefined position in the modified bitstream and/orincludes a specific instance tag.

10. Method according to A-EEE 6 or 7, wherein the MPEG-4 audio bitstreamextension mechanism is a Fill Element (ID_FIL).

A-EEE 11. Method according to A-EEE 10, wherein the generated payload istransported via a generic extension payload element inside the FillElement.

A-EEE 12. Method according to A-EEE 11, wherein further the generatedinformation and/or the generated information relating to the relativegain are transported via the generic extension payload element insidethe Fill Element.

A-EEE 13. Method according to A-EEE 10, wherein the generated payload istransported via a new extension payload element inside the Fill Element.

A-EEE 14. Method according to A-EEE 13, wherein further the generatedinformation and/or the generated information relating to the relativegain are transported via the new extension payload element inside theFill Element.

A-EEE 15. Method according to A-EEE 13 or 14, wherein the new extensionpayload element includes a unique identifier.

A-EEE 16. Method according to any of A-EEEs 1 to 15, wherein in step c)the modified bitstream is output to the sink device via a Bluetoothconnection.

A-EEE 17. Method according to any of A-EEEs 1 to 16, wherein theadditional media data are compressed media data or uncompressed mediadata.

A-EEE 18. Method according to A-EEE 17, wherein the uncompressed mediadata are PCM data generated at the source device.

A-EEE 19. Method according to A-EEE 18, wherein the generation of thePCM data is based on a user input.

A-EEE 20. Method according to A-EEE 18 or 19, wherein the PCM datainclude one or more of earcon-data and system sounds data.

A-EEE 21. Method according to A-EEE 17, wherein the compressed data arein SBC format or in an aptX-based format.

A-EEE 22. A method for processing a modified bitstream on a sink device,wherein the method includes the steps of:

-   -   a) receiving, by a receiver, a modified bitstream including        coded media data and payload of additional media data;    -   b) parsing, by a bitstream parser, the modified bitstream into        the coded media data and the payload of additional media data;    -   c) core decoding, by a core decoder, the coded media data to        obtain core decoded media data;    -   d) mixing, by a mixer, the core decoded media data and the        additional media data to obtain an output signal; and    -   e) outputting the output signal.

A-EEE 23. Method according to A-EEE 22, wherein the modified bitstreamfurther includes information on the additional media data, and whereinthe method further includes, after step a) and before step b), a step ofprocessing the modified bitstream based on the information.

A-EEE 24. Method according to A-EEE 23, wherein the information includesone or more of information on a configuration of the additional mediadata, information on a presence of the additional media data in themodified bitstream, information on a default loudness value, informationon a value for delay alignment, information on delay adaption,information on panning and information on a type of the additional mediadata.

A-EEE 25. Method according to any of A-EEEs 22 to 24, wherein themodified bitstream further includes information relating to a relativegain between the coded media data and the additional media data, andwherein mixing in step d) the core decoded media data and the additionalmedia data is based on the information relating to the relative gain.

A-EEE 26. Method according to any of A-EEEs 22 to 25, wherein the methodfurther includes the step of processing, by a processing unit, theadditional media data prior to mixing the core decoded media data andthe additional media data.

A-EEE 27. Method according to A-EEE 26, wherein processing theadditional media data includes one or more of resampling, delay adaptionand loudness processing.

A-EEE 28. Method according to any of A-EEEs 22 to 27, wherein theadditional media data are compressed media data and the method furtherincludes the step of decoding, by a decoder, the compressed media datato obtain decoded additional media data, and wherein in step d) thedecoded additional media data are mixed with the core decoded mediadata.

A-EEE 29. Method according to A-EEE 28, wherein the compressed mediadata are in SBC format or in an aptX-based format.

A-EEE 30. Method according to any of A-EEEs 22 to 27, wherein theadditional media data are uncompressed media data, and wherein in stepd) the uncompressed additional media data are mixed with the coredecoded media data.

A-EEE 31. Method according to any of A-EEEs 22 to 30, wherein themodified bitstream is in MPEG-4 audio format.

A-EEE 32. Method according to A-EEE 31, wherein the modified bitstreamincludes an MPEG-4 audio bitstream extension mechanism, and whereinparsing in step b) is further based on identifying the MPEG-4 audiobitstream extension mechanism in the modified bitstream.

A-EEE 33. Method according to A-EEE 32, wherein the MPEG-4 audiobitstream extension mechanism is a data stream element (DSE) and parsingin step b) is further based on identifying a position of the data streamelement in the modified bitstream and/or identifying a specific instancetag of the data stream element.

A-EEE 34. Method according to A-EEE 32, wherein the MPEG-4 audiobitstream extension mechanism is a Fill Element (ID_FIL) and parsing instep b) is further based on identifying an extension payload elementinside the Fill Element.

A-EEE 35. Source device for generating a modified bitstream, wherein thedevice includes:

-   -   a) a receiver configured to receive a bitstream including coded        media data;    -   b) an embedder configured to generate payload of additional        media data and to embed the generated payload in the bitstream        to obtain a modified bitstream including the coded media data        and the payload of the additional media data; and    -   c) a control unit configured to output the modified bitstream to        a sink device.

A-EEE 36. Device according to A-EEE 35, wherein the embedder is furtherconfigured to generate information on the additional media data and toembed the information in the bitstream.

A-EEE 37. Device according to A-EEE 35 or 36, wherein the embedder isfurther configured to generate information relating to a relative gainbetween the coded media data and the additional media data and to embedthe information relating to the relative gain in the bitstream.

A-EEE 38. Device according to any of A-EEEs 35 to 37, wherein themodified bitstream is in MPEG-4 audio format.

A-EEE 39. Device according to any of A-EEEs 35 to 38, wherein theadditional media data are compressed media data or uncompressed mediadata.

A-EEE 40. Device according to A-EEE 39, wherein the uncompressed mediadata are PCM data generated at the source device.

A-EEE 41. Device according to A-EEE 40, wherein the generation of thePCM data is based on a user input.

A-EEE 42. Device according to A-EEE 40 or 41, wherein the PCM datainclude one or more of earcon-data and system sounds data.

A-EEE 43. Device according to A-EEE 39, wherein the compressed mediadata are in SBC format or in an aptX-based format.

A-EEE 44. Sink device for processing a modified bitstream, wherein thedevice includes:

-   -   a) a receiver configured to receive a modified bitstream        including coded media data and payload of additional media data;    -   b) a bitstream parser configured to parse the modified bitstream        into the coded media data and the payload of the additional        media data;    -   c) a core decoder configured to core decode the coded media data        to obtain core decoded media data;    -   d) a mixer configured to mix the core decoded media data and the        additional media data to obtain an output signal; and    -   e) a control unit configured to output the output signal.

A-EEE 45. Device according to A-EEE 44, wherein the device furtherincludes a processing unit configured to process the additional mediadata prior to mixing the core decoded media data and the additionalmedia data.

A-EEE 46. Device according to A-EEE 44 or 45, wherein the additionalmedia data are uncompressed media data, and wherein the mixer isconfigured to mix the core decoded media data and the uncompressedadditional media data.

A-EEE 47. Device according to A-EEE 44 or 45, wherein the additionalmedia data are compressed media data and the device further includes adecoder to decode the compressed media data to obtain decoded additionalmedia data, and wherein the mixer is configured to mix the core decodedmedia data and the decoded additional media data.

A-EEE 48. Device according to any of A-EEEs 44 to 47, wherein themodified bitstream further includes information on the additional mediadata, and wherein the sink device is configured to process the modifiedbitstream based on the information.

A-EEE 49. Device according to any of A-EEEs 44 to 48, wherein themodified bitstream further includes information relating to a relativegain between the coded media data and the additional media data, andwherein the mixer is further configured to mix the core decoded mediadata and the additional media data based on the information relating tothe relative gain.

A-EEE 50. Device according to any of A-EEEs 44 to 49, wherein the coredecoder is an MPEG-4 audio decoder.

A-EEE 51. System of a source device for generating a modified bitstream,wherein the source device includes one or more processors configured toperform the method according to any of A-EEEs 1 to 21 and a sink devicefor processing a modified bitstream, wherein the sink device includesone or more processors configured to perform the method according toA-EEEs 22 to 34.

A-EEE 52. A computer program product comprising a computer-readablestorage medium with instructions adapted to cause the device to carryout the method according to A-EEE 1 when executed by a device havingprocessing capability.

A-EEE 53. A computer program product comprising a computer-readablestorage medium with instructions adapted to cause the device to carryout the method according to A-EEE 22 when executed by a device havingprocessing capability.

B-EEE 1. A method for generating a modified bitstream on a sourcedevice, wherein the method includes the steps of:

-   -   a) receiving, by a receiver, a bitstream including coded media        data;    -   b) generating, by an embedder, payload of additional media data        and embedding the payload in the bitstream for obtaining, as an        output from the embedder, a modified bitstream including the        coded media data and the payload of the additional media data;        and    -   c) outputting the modified bitstream to a sink device.

B-EEE 2. Method according to B-EEE 1,

-   -   wherein step b) further includes generating information on the        additional media data and embedding the information in the        bitstream; and optionally    -   wherein the generated information includes one or more of        information on a configuration of the additional media data,        information on a presence of the additional media data in the        modified bitstream, information on a default loudness value,        information on a value for delay alignment, information on delay        adaption, information on panning and information on a type of        the additional media data.

B-EEE 3. Method according to B-EEE 1, wherein step b) further includesgenerating information relating to a relative gain between the codedmedia data and the additional media data and embedding the informationrelating to the relative gain in the bitstream.

B-EEE 4. Method according to B-EEE 1,

-   -   wherein the modified bitstream is in MPEG-4 audio format; and    -   wherein the generated payload is embedded in the bitstream by        transporting the payload in the bitstream via an MPEG-4 audio        bitstream extension mechanism.

B-EEE 5. Method according to B-EEE 3,

-   -   wherein the modified bitstream is in MPEG-4 audio format; and    -   wherein the generated information and/or the generated        information relating to the relative gain are embedded in the        bitstream by transporting the generated information and/or the        generated information relating to the relative gain in the        bitstream via an MPEG-4 audio bitstream extension mechanism.

B-EEE 6. Method according to B-EEE 4,

-   -   wherein the MPEG-4 audio bitstream extension mechanism is a data        stream element (DSE); or    -   wherein the MPEG-4 audio bitstream extension mechanism is a Fill        Element (ID_FIL), and wherein optionally the generated payload        is transported via a generic extension payload element or a new        extension payload element inside the Fill Element.

B-EEE 7. Method according to B-EEE 1, wherein in step c) the modifiedbitstream is output to the sink device via a Bluetooth connection.

B-EEE 8. Method according to B-EEE 1, wherein the additional media dataare PCM data generated at the source device.

B-EEE 9. Method according to B-EEE 8, wherein the PCM data include oneor more of earcon-data and system sounds data.

B-EEE 10. Method according to B-EEE 1, wherein the additional media dataare compressed data in SBC format or in an aptX-based format.

B-EEE 11. A method for processing a modified bitstream on a sink device,wherein the method includes the steps of:

-   -   a) receiving, by a receiver, a modified bitstream including        coded media data and payload of additional media data;    -   b) parsing, by a bitstream parser, the modified bitstream into        the coded media data and the payload of additional media data;    -   c) core decoding, by a core decoder, the coded media data to        obtain core decoded media data;    -   d) mixing, by a mixer, the core decoded media data and the        additional media data to obtain an output signal; and    -   e) outputting the output signal.

B-EEE 12. Method according to B-EEE 11, wherein the modified bitstreamfurther includes information relating to a relative gain between thecoded media data and the additional media data, and wherein mixing instep d) the core decoded media data and the additional media data isbased on the information relating to the relative gain.

B-EEE 13. Method according to B-EEE 11,

-   -   wherein the method further includes the step of processing, by a        processing unit, the additional media data prior to mixing the        core decoded media data and the additional media data; and        optionally    -   wherein processing the additional media data includes one or        more of resampling, delay adaption and loudness processing.

B-EEE 14. Method according to B-EEE 11,

-   -   wherein the modified bitstream is in MPEG-4 audio format; and    -   wherein the modified bitstream includes an MPEG-4 audio        bitstream extension mechanism, and wherein parsing in step b) is        further based on identifying the MPEG-4 audio bitstream        extension mechanism in the modified bitstream.

B-EEE 15. Method according to B-EEE 14,

-   -   wherein the MPEG-4 audio bitstream extension mechanism is a data        stream element (DSE) and parsing in step b) is further based on        identifying a position of the data stream element in the        modified bitstream and/or identifying a specific instance tag of        the data stream element; or    -   wherein the MPEG-4 audio bitstream extension mechanism is a Fill        Element (ID_FIL) and parsing in step b) is further based on        identifying an extension payload element inside the Fill        Element.

B-EEE 16. Source device for generating a modified bitstream, wherein thedevice includes:

-   -   a) a receiver configured to receive a bitstream including coded        media data;    -   b) an embedder configured to generate payload of additional        media data and to embed the generated payload in the bitstream        to obtain a modified bitstream including the coded media data        and the payload of the additional media data; and    -   c) a control unit configured to output the modified bitstream to        a sink device.

B-EEE 17. Sink device for processing a modified bitstream, wherein thedevice includes:

-   -   a) a receiver configured to receive a modified bitstream        including coded media data and payload of additional media data;    -   b) a bitstream parser configured to parse the modified bitstream        into the coded media data and the payload of the additional        media data;    -   c) a core decoder configured to core decode the coded media data        to obtain core decoded media data;    -   d) a mixer configured to mix the core decoded media data and the        additional media data to obtain an output signal; and    -   e) a control unit configured to output the output signal.

B-EEE 18. System of a source device for generating a modified bitstream,wherein the source device includes one or more processors configured toperform the method according to B-EEE 1 and a sink device for processinga modified bitstream, wherein the sink device includes one or moreprocessors configured to perform the method according to B-EEE 11.

B-EEE 19. A computer program product comprising a computer-readablestorage medium with instructions adapted to cause the device to carryout the method according to B-EEE 1 when executed by a device havingprocessing capability.

B-EEE 20. A computer program product comprising a computer-readablestorage medium with instructions adapted to cause the device to carryout the method according to B-EEE 11 when executed by a device havingprocessing capability.

1. A method for generating a modified bitstream in MPEG-4 audio formatby a source device, the source device comprising a receiver and anembedder, wherein the method includes the steps of: a) receiving, by thereceiver, a bitstream including coded media data; b) generating, by theembedder, payload of additional uncompressed PCM audio data andembedding the payload in the bitstream for obtaining, as an output fromthe embedder, a modified bitstream including the coded media data andthe payload of the additional uncompressed PCM audio data, wherein thegenerated payload is embedded in the bitstream by transporting thepayload in the modified bitstream via an MPEG-4 audio bitstreamextension mechanism, wherein the MPEG-4 audio bitstream extensionmechanism is a data stream element (DSE); and c) outputting the modifiedbitstream to the sink device.
 2. Method according to claim 1, whereinstep b) further includes generating information on the additionaluncompressed PCM audio data and embedding the information in thebitstream.
 3. Method according to claim 2, wherein the generatedinformation includes one or more of information on a configuration ofthe additional uncompressed PCM audio data, information on a presence ofthe additional uncompressed PCM audio data in the modified bitstream,information on a default loudness value, information on a value fordelay alignment, information on delay adaption, information on panningand information on a type of the additional data uncompressed PCM audiodata.
 4. Method according to claim 1, wherein step b) further includesgenerating information relating to a relative gain between the codedmedia data and the additional uncompressed PCM audio data and embeddingthe information relating to the relative gain in the bitstream. 5.(canceled)
 6. (canceled)
 7. Method according to claim 2, wherein furtherthe generated information and/or the generated information relating tothe relative gain are embedded in the bitstream by transporting thegenerated information and/or the generated information relating to therelative gain in the modified bitstream via the MPEG-4 audio bitstreamextension mechanism.
 8. (canceled)
 9. Method according to claim 1,wherein the MPEG-4 audio bitstream extension mechanism is a data streamelement (DSE) and the data stream element (DSE) is located at apredefined position in the modified bitstream and/or includes a specificinstance tag. 10-15. (canceled)
 16. Method according to claim 1, whereinin step c) the modified bitstream is output to the sink device via aBluetooth connection. 17-18. (canceled)
 19. Method according to claim 1,wherein the generation of the PCM audio data is based on a user input.20. Method according to claim 1, wherein the PCM audio data include oneor more of earcon-data and system sounds data.
 21. (canceled)
 22. Amethod for processing a modified bitstream in MPEG-4 audio format by asink device comprising a receiver, a bitstream parser, a core decoder, amixer, wherein the method includes the steps of: a) receiving, by thereceiver, a modified bitstream including coded media data, payload ofadditional uncompressed PCM audio data and an MPEG-4 audio bitstreamextension mechanism, wherein the MPEG-4 audio bitstream extensionmechanism is a data stream element (DSE); b) parsing, by the bitstreamparser, the modified bitstream into the coded media data and the payloadof additional uncompressed PCM audio data, wherein the parsing is basedon identifying the MPEG-4 audio bitstream extension mechanism in themodified bitstream; c) core decoding, by the core decoder, the codedmedia data to obtain core decoded media data; d) mixing, by the mixer,the core decoded media data and the additional uncompressed PCM audiodata to obtain an output signal; and e) outputting the output signal.23. Method according to claim 22, wherein the modified bitstream furtherincludes information on the additional uncompressed PCM audio data, andwherein the method further includes, after step a) and before step b), astep of processing the modified bitstream based on the information. 24.Method according to claim 23, wherein the information includes one ormore of information on a configuration of the additional uncompressedPCM audio data, information on a presence of the additional uncompressedPCM audio data in the modified bitstream, information on a defaultloudness value, information on a value for delay alignment, informationon delay adaption, information on panning and information on a type ofthe additional uncompressed PCM audio data.
 25. Method according toclaim 22, wherein the modified bitstream further includes informationrelating to a relative gain between the coded media data and theadditional uncompressed PCM audio data, and wherein mixing in step d)the core decoded media data and the additional uncompressed PCM audiodata is based on the information relating to the relative gain. 26.Method according to claim 10, wherein the method further includes thestep of processing, by a processing unit, the additional uncompressedPCM audio data prior to mixing the core decoded media data and theadditional uncompressed PCM audio data.
 27. Method according to claim26, wherein processing the additional uncompressed PCM audio dataincludes one or more of resampling, delay adaption and loudnessprocessing. 28-32. (canceled)
 33. Method according to claim 22, whereinparsing in step b) is further based on identifying a position of thedata stream element in the modified bitstream and/or identifying aspecific instance tag of the data stream element.
 34. (canceled) 35.Source device for generating a modified bitstream in MPEG-4 audioformat, wherein the device includes: a) a receiver configured to receivea bitstream including coded media data; b) an embedder configured togenerate payload of additional uncompressed PCM audio data and to embedthe generated payload in the bitstream to obtain a modified bitstreamincluding the coded media data and the payload of the additionaluncompressed PCM audio data wherein the embedder is further configuredto embed the generated payload in the bitstream by transporting thepayload in the modified bitstream via an MPEG-4 audio bitstreamextension mechanism, wherein the MPEG-4 audio bitstream extensionmechanism is a data stream element (DSE); and c) a control unitconfigured to output the modified bitstream to a sink device. 36-43.(canceled)
 44. Sink device for processing a modified bitstream in MPEG-4audio format, wherein the device includes: a) a receiver configured toreceive a modified bitstream including coded media data, payload ofadditional uncompressed PCM audio data and an MPEG-4 audio bitstreamextension mechanism, wherein the MPEG-4 audio bitstream extensionmechanism is a data stream element (DSE); b) a bitstream parserconfigured to parse the modified bitstream into the coded media data andthe payload of the additional uncompressed PCM audio data, wherein thebitstream parser is further configured to parse the modified bitstreambased on identifying the MPEG-4 audio bitstream extension mechanism inthe modified bitstream; c) a core decoder configured to core decode thecoded media data to obtain core decoded media data; d) a mixerconfigured to mix the core decoded media data and the additionaluncompressed PCM audio data to obtain an output signal; and e) a controlunit configured to output the output signal.
 45. Device according toclaim 44, wherein the device further includes a processing unitconfigured to process the additional uncompressed PCM audio data priorto mixing the core decoded media data and the additional uncompressedPCM audio data. 46-47. (canceled)
 48. Device according to claim 48,wherein the modified bitstream further includes information on theadditional uncompressed PCM audio data, and wherein the sink device isconfigured to process the modified bitstream based on the information.49. Device according to claim 48, wherein the modified bitstream furtherincludes information relating to a relative gain between the coded mediadata and the additional uncompressed PCM audio data, and wherein themixer is further configured to mix the core decoded media data and theadditional uncompressed PCM audio data based on the information relatingto the relative gain.
 50. Device according to claim 48, wherein the coredecoder is an MPEG-4 audio decoder. 51-52. (canceled)